Object extraction techniques and visual image search
with Semantic web techniques
Aninda Maulik
Supervisors: Prof. Pierre Maret
Dennis Diefenbach
Cyber Physical and Social Systems
University of Jean Monnet
July 2020
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 1 / 42
Introduction:QAnswer
QAnswer is knowledge based QA system.
A knowledge base is a collection of facts that can be interpreted by a
machine
Such a fact can look like this:”bicycle” ”in” ”right”.
QAnswer translates from a natural language question to correct
SPARQL queries.
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 4 / 42
Content
Implementation of an Algorithm for object extraction.
Design of a semantic web modelling for extracted data.
Implementation of a visual image search engine through Qanswer.
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 5 / 42
After Semantic Web Modelling
1
We convert the CSV file created for Image-Object Relations into
a RDF file
with the help of a Java Program
2
and upload the RDF file into QAnswer
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 29 / 42
special Wikimediacommons api give images and human
hand-annotated structured data
The Api
https://commons.wikimedia.org/w/api.php?action=query&list=
search&srsearch=haswbstatement:
P180=Q7378&srnamespace=6&format=json
give images in form of json data
and human hand-annotated structured data for some images
Examples
1
what are the items portrayed in the image
2
who is the creator of the file
3
what is the source of the file
4
inception date of the file
5
copyright status
6
The url showing structured data
https://commons.wikimedia.org/wiki/File:
African_elephants,_Lake_St_Lucia_06.jpg
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 34 / 42
special Wikimediacommons api give images and human
hand annotated structured data
Our time constraints
https://commons.wikimedia.org/w/api.php?action=query&list=
search&srsearch=haswbstatement:
P180=Q7378&srnamespace=6&format=json
We are yet to handle the incoming RDF data from the api
and merge with our RDF file,
in order to be able to query
”Give me pictures of bicycles from February 2019”
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 40 / 42
Conclusion
1
We have worked on improving image search engines
2
Computer Vision techniques is able to identify objects in images.
3
Semantic Web techniques give a semantic representation of the
images that can be queried QAnswer.
4
This work can be easily used by any search or query engine to give
results
5
We do not have a bench mark as of yet
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 41 / 42
